'ZIPLIB' COMPRESSED DATA FORMAT SPECIFICATION draft #3 March 7, 1995 Copyright (C) 1995 L. Peter Deutsch and Jean-loup Gailly Permission is granted to copy and distribute this document for any purpose and without charge, provided that it is copied as a whole (including the copyright notice and this notice) and with no changes. 1. Introduction 1.1 Purpose The purpose of this specification is to define a lossless compressed data format that: (a) Is independent of CPU type, operating system, file system, and character set, and hence can be used for interchange; (b) Can be produced or consumed, even for an arbitrarily long sequentially presented input data stream, using only an a priori bounded amount of intermediate storage, and hence can be used in data communications or similar structures such as Unix filters; (c) Can use a number of different compression methods; (d) Can be implemented readily in a manner not covered by patents, and hence can be practiced freely. The data format defined by this specification does not attempt to allow random access to compressed data. 1.2 Intended audience This specification is intended for use by implementors of software to compress data into ziplib format and/or decompress data from ziplib format. The text of the specification assumes a basic background in programming at the level of bits and other primitive data representations. 1.3 Scope The specification specifies a compressed data format, to be used for in-memory compression of a sequence of arbitrary octets. 1.4 Compliance Unless otherwise indicated below, a compliant decompressor must be able to accept and decompress any data set that conforms to all the specifications presented here; a compliant compressor must produce data sets that conform to all the specifications presented here. 1.4 Related standards None. 1.5 Other related publications Deutsch, L.P.,"'Gzip' Compressed Data Format Specification". Drafts being circulated. Deutsch, L.P.,"'Deflate' Compressed Data Format Specification". Drafts being circulated. Thomas Boutell, "PNG (Portable Network Graphics) specification". Fletcher, J. G., "An Arithmetic Checksum for Serial Transmissions," IEEE Transactions on Communications, Vol. COM-30, No. 1, January 1982, pp. 247-252. ITU-T Recommendation X.224, Annex D, "Checksum Algorithms," November, 1993, pp. 144, 145. (Available from gopher://info.itu.ch). ITU-T X.244 is also the same as ISO 8073. 1.6 Definitions of terms and conventions used octet: 8 bits stored or transmitted as a unit (same as a byte on most machines). See section 3.1 below for the numbering of bits within an octet. 2. Detailed specification. 2.1 Overall conventions. In the diagrams below, a box like this: +---+ | | <-- the vertical bars might be missing +---+ represents one octet; a box like this: +==============+ | | +==============+ represents a variable number of octets. Octets stored within a computer do not have a 'bit order', since they are always treated as a unit. However, an octet considered as an integer between 0 and 255 does have a most- and least-significant bit, and since we write numbers with the most-significant digit on the left, we also write octets with the most-significant bit on the left. In the diagrams below, we number the bits of an octet so that bit 0 is the least-significant bit, i.e., the bits are numbered: +--------+ |76543210| +--------+ Within a computer, a number may occupy multiple octets. All multi-octet numbers in the format described here are stored with the MOST-significant octet first (at the lower memory address). For example, the decimal number 520 is stored as: 0 1 +--------+--------+ |00000010|00001000| +--------+--------+ ^ ^ | | | + less significant octet = 8 + more significant octet = 2 x 256 2.2 Data format. The data have the following structure: 0 1 +---+---+=====================+---+---+---+---+ |CMF|FLG|...compressed data...| ADLER32 | +---+---+=====================+---+---+---+---+ CMF (Compression Method and flags) This octet is divided into a 4-bit compression method and a 4-bit information field depending on the compression method. bits 0 to 3 CM Compression method bits 4 to 7 CINFO Compression info CM (Compression method) This identifies the compression method used in the file. CM = 8 denotes the 'deflate' compression method with a window size up to 32K. This is the method used by gzip; it is documented elsewhere. CINFO (Compression info) For CM = 8, CINFO is the base-2 logarithm of the LZ77 window size, minus eight (CINFO=7 indicates a 32K window size). Values of CINFO above 7 are not allowed in this version of the specification. CINFO is not defined in this specification for CM not equal to 8. FLG (FLaGs) This flag octet is divided as follows: bits 0 to 4 FCHECK (check bits for CMF and FLG) bit 5 reserved, must be zero bits 6 to 7 FLEVEL (compression level) The FCHECK value must be such that CMF and FLG, when viewed as a 16-bit unsigned integer stored in MSB order (CMF*256 + FLG), is a multiple of 31. FLEVEL (Compression level) These flags are available for use by specific compression methods. The 'deflate' method (CM = 8) sets these flags as follows: 0 - compressor used fastest algorithm 1 - compressor used fast algorithm 2 - compressor used default algorithm 3 - compressor used maximum compression, slowest algorithm The information in FLEVEL is not needed for decompression; it is there to indicate if recompression might be worthwhile. ADLER32 (Adler-32 cheksum) This contains a checksum value of the uncompressed data computed according to Adler-32 algorithm. This algorithm is a 32-bit extension and improvement of the Fletcher algorithm, used in the ITU-T X.224 / ISO 8073 standard. Adler-32 is composed of two sums accumulated per octet: s1 is the sum of all octets, s2 is the sum of all s1 values. Both sums are done modulo 65521. s1 is initialized to 1, s2 to zero. The Adler-32 checksum is stored as s2*65536 + s1 in most-significant-octet first (network) order. 3. Appendix: Rationale The Adler-32 algorithm is much faster than the CRC32 algorithm yet still provides an extremely low probability of undetected errors. The modulo on unsigned long accumulators can be delayed for 5552 octets, so the modulo operation time is negligible. If the octets are a, b, c, the second sum is 3a + 2b + c + 3, and so is position and order sensitive, unlike the first sum, which is just a checksum. That 65521 is prime is important to avoid a possible large class of two-octet errors that leave the check unchanged. (The Fletcher checksum uses 255, which is not prime and which also makes the Fletcher check insensitive to single octet changes 0 <-> 255.) The sum s1 is initialized to 1 instead of zero to make the length of the sequence part of s2, so that the length does not have to be checked separately. (Any sequence of zeroes has a Fletcher checksum of zero.) The following C code computes the Adler-32 checksum of a data buffer: #define BASE 65521 /* largest prime smaller than 65536 */ #define NMAX 5552 /* NMAX is the largest n such that 255n(n+1)/2 + (n+1)(BASE-1) <= 2^32-1 */ unsigned long adler32(unsigned char *b, int n) { unsigned char *p = b; unsigned long s1 = 1; unsigned long s2 = 0; int k; while (n > 0) { k = n < NMAX ? n : NMAX; n -= k; do { /* could unroll loop some for speed */ s1 += *p++; s2 += s1; } while (--k); s1 %= BASE; s2 %= BASE; } return (s2 << 16) | s1; }